Systems and methods for predicting control handback

ABSTRACT

Embodiments described herein include systems and methods for predicting a transfer of control of a vehicle to a driver. A method includes receiving information about an environment of the vehicle, identifying at least one condition represented in the information about the environment of the vehicle that corresponds to at least one of one or more known conditions that lead to a handback of operational control of the vehicle to the driver, and predicting the transfer of control of the vehicle to the driver based on the at least one condition identified from the information about the environment of the vehicle.

TECHNICAL FIELD

The present specification generally relates to systems and methods forpredicting the transfer of operational control of a vehicle to a driver.

BACKGROUND

Autonomous operation of vehicles continues to evolve and is implementedin varying degrees to assist a driver with operational control of avehicle. Autonomous systems, for example, provide assistive measures,such as braking, limited full control of vehicle maneuvers whentraveling in particular environments, full autonomous control of vehiclemaneuvers when traveling between locations, and other levels of controltherebetween. Moreover, while autonomous systems continue to develop,many autonomous systems require human interaction during portions of adrive. As a result, a vehicle may handback control of a vehicle to ahuman driver from time to time during the drive. However, effectivehandback of control to a human driver is fraught with issues. Theseissues include, for example, the inability to predict when a handbackevent will occur, the inability to effectively make the human driveraware of an impending handback of control, and the inability tosuccessfully provide the human driver with information about theenvironment and state of the vehicle before operational control of thevehicle is transferred to the human driver.

SUMMARY

In embodiments, a method for predicting a transfer of control of avehicle to a driver includes receiving information about an environmentof the vehicle, identifying at least one condition represented in theinformation about the environment of the vehicle that corresponds to atleast one of one or more known conditions that lead to a handback ofoperational control of the vehicle to the driver, and predicting thetransfer of control of the vehicle to the driver based on the at leastone condition identified from the information about the environment ofthe vehicle.

In some embodiments, a system for predicting a transfer of control of avehicle to a driver includes an electronic control unit and one or moreenvironment sensors communicatively coupled to the electronic controlunit. The one or more environment sensors capture information about anenvironment of the vehicle. The electronic control unit is configured toreceive the information about the environment of the vehicle from theone or more environment sensors, identify at least one conditionrepresented in the information about the environment of the vehicle thatcorresponds to at least one of one or more known conditions that lead toa handback of operational control of the vehicle to the driver, andpredict the transfer of control of the vehicle to the driver based onthe at least one condition identified from the information about theenvironment of the vehicle.

In some embodiments, a system for predicting a transfer of control of avehicle to a driver includes an electronic control unit configured toimplement a neural network and one or more environment sensorscommunicatively coupled to the electronic control unit. The one or moreenvironment sensors capture information about an environment of thevehicle. The electronic control unit is configured to receive, as aninput to the neural network, the information about the environment ofthe vehicle from the one or more environment sensors, identify, with theneural network, at least one condition represented in the informationabout the environment of the vehicle that corresponds to at least one ofone or more known conditions that lead to a handback of operationalcontrol of the vehicle to the driver, and predict, with the neuralnetwork, the transfer of control of the vehicle to the driver based onthe at least one condition identified from the information about theenvironment of the vehicle.

These and additional features provided by the embodiments describedherein will be more fully understood in view of the following detaileddescription, in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the subject matter defined by theclaims. The following detailed description of the illustrativeembodiments can be understood when read in conjunction with thefollowing drawings, where like structure is indicated with likereference numerals and in which:

FIG. 1 depicts an illustrative system for predicting a transfer ofcontrol of a vehicle to a driver, according to one or more embodimentsshown and described herein;

FIG. 2 depicts an illustrative vehicle implemented with the systemdepicted in FIG. 1, according to one or more embodiments shown anddescribed herein;

FIG. 3 depicts another illustrative environment for capturing a gaze ofa driver utilizing a gaze tracking system, according to one or moreembodiments shown and described herein;

FIG. 4 depicts an illustrative diagram for a neural network model forpredicting the transfer of control of the vehicle to the driver,according to one or more embodiments shown and described herein;

FIG. 5A depicts an illustrative flow diagram predicting a transfer ofcontrol of a vehicle to a driver, according to one or more embodimentsshown and described herein;

FIG. 5B further depicts the illustrative flow diagram predicting atransfer of control of a vehicle to a driver, according to one or moreembodiments shown and described herein;

FIG. 5C further depicts the illustrative flow diagram predicting atransfer of control of a vehicle to a driver, according to one or moreembodiments shown and described herein;

FIG. 6A depicts an illustrative environment around a vehicle includingone or more conditions indicative of a transfer of control of thevehicle to the driver, according to one or more embodiments shown anddescribed herein;

FIG. 6B depicts an illustrative environment around a vehicle includingsalient portions of the environment identifying the one or more knownconditions indicative of a transfer of control of the vehicle to thedriver, according to one or more embodiments shown and described herein;and

FIG. 6C depicts an illustrative environment around a vehicle includinggaze pattern of a driver, according to one or more embodiments shown anddescribed herein.

DETAILED DESCRIPTION

Embodiments described herein relate to systems and methods forpredicting occurrences of handback events (i.e., the transfer ofoperational control of a vehicle to a driver). Handback events mayoccur, for example, when an autonomous system has completed a routinefor assisting a driver to navigate a vehicle, when the autonomous systemencounters an environment or situation which the system is not capableof navigating or may require intermittent or full assumption of controlby a driver, or when a driver decides to retake control of a vehicle.Regardless of the reason why a handback event is triggered, it isimportant that a driver be prepared to assume control. The systems andmethods disclosed herein are directed to predicting an occurrence of ahandback event, which may allow a driver may be duly informed andprepared to assume control. For example, as autonomous systems advancethere will be fewer and fewer situations where a driver needs tomaintain full attention to the driving environment. However, when asituation arises that may require the driver's engagement in control ofthe vehicle, a driver may need to be alerted with sufficient notice toprepare to assume control.

One of the challenges in handing back control of a vehicle isdetermining when these events will occur. The challenge includes notonly determining whether the driver is ready and aware to receiveoperational control of the vehicle, but also in providing an alert thatwill not be ignored over time due to false positive activation. As such,depending on how likely a handback event is to occur in the future, adegree or type of alert may be determined and generated that correspondsto the likelihood and/or imminence of the predicted handback event. Insome embodiments, an aspect of the systems and methods disclosed hereinis to provide a degree or type of alert that correspond to thelikelihood and/or imminence of the predicted handback event. In someembodiments, an aspect of the systems and methods disclosed herein is toprovide an alert that corresponds to the driver's attention and/orawareness to the present driving environment. For example, if the driveris well aware of the environment, a subtle alert may be provided.Whereas, if the driver is distracted, for example, looking at theirphone and not the road while the vehicle is in an autonomous mode, thena progressively increasing degree or various types of alerts (e.g.,audio, visual, haptic, or the like) may need to be implemented.

Embodiments described in more detail herein generally include utilizinginformation generated by environment sensors configured to capturefeatures and the environment around a vehicle to predict whether ahandback event will occur. The information may include road scene datain the form of image data, LIDAR data, or another form of mapping datathat can be analyzed to determine conditions within the environment. Asused herein, the term “conditions” refers to the detection of an object,sign, vehicle, animal, person, or the like within the environment whichwhen combined with other circumstances may form a sequence ofoccurrences or events that may lead to a handback event. For example,detection that traffic is slowing down may be a first condition,detection of a construction sign may be another condition, and detectionof a flagman directing traffic may be yet another condition. As each ofthese conditions are detected, a prediction that a handback event willoccur may be made. Moreover, as more conditions are detected andcompared with known conditions that have led to past handback events,the likelihood that a handback event will occur may increase.

Referring back to the aforementioned example, when taken in combination,these conditions may indicate that there is an approaching detour in theroadway. In the event a detour is determined, an autonomous system mayrequire a driver begin paying closer attention to the environment sothat she may readily assume control if necessary or receive operationalcontrol and navigate through the detour. An alert having a degree ortype corresponding to the imminence and/or likelihood of the predictedhandback event may be implemented to communicate with the driver andprepare the driver for a possible transfer of operational control froman autonomous system to the driver. This is only one non-limitingexample of detected conditions within an environment that result in aprediction of a handback event.

As will be described in more detail herein, in some embodiments, thesystems and methods may implement a neural network that is configured tolearn one or more conditions and/or sequences of conditions that lead toa handback event. When trained, the neural network may receiveinformation from environment sensors and predict of occurrences ofhandback events.

Embodiments of the present disclosure are directed to systems andmethods for predicting occurrences of a handback event and providingalerts to a driver based on the predictions. The following will nowdescribe these systems and methods in more detail with reference to thedrawings and where like numbers refer to like structures.

Referring now to FIG. 1, a system 100 for predicting a transfer ofcontrol of a vehicle to a driver is depicted. The system 100 may includean electronic control unit 130. The electronic control unit 130 mayinclude a processor 132 and a memory component 134. The system 100 mayalso include a communication bus 120, a LIDAR system 136, one or morecameras 138, a gaze-tracking system 140, an illuminating device 141, oneor more physiological sensors 142, a speaker 144, a steering wheelsystem 146, a heads-up display system 148, a vehicle display 149, a datastorage component 150 and/or network interface hardware 170. As referredto herein, the term “one or more environment sensors” may include theLIDAR system 136, one or more cameras 138, and/or a variety of othersensor systems capable of ascertaining information about the environmentaround a vehicle and functionality of the vehicle such as a vehiclespeed, a rate of acceleration or deceleration of the vehicle, a vehiclelocation, a vehicle heading, or the like. The system 100 may becommunicatively coupled to a network 180 by way of the network interfacehardware 170. The components of the system 100 are communicativelycoupled to each other via the communication bus 120.

It is understood that the embodiments depicted and described herein arenot limited to the components or configurations depicted and describedwith respect to FIG. 1, rather FIG. 1 is merely for illustration. Thevarious components of the system 100 and the interaction thereof will bedescribed in detail below.

The communication bus 120 may be formed from any medium that is capableof transmitting a signal such as, for example, conductive wires,conductive traces, optical waveguides, or the like. The communicationbus 120 may also refer to the expanse in which electromagnetic radiationand their corresponding electromagnetic waves traverses. Moreover, thecommunication bus 120 may be formed from a combination of mediumscapable of transmitting signals. In one embodiment, the communicationbus 120 comprises a combination of conductive traces, conductive wires,connectors, and buses that cooperate to permit the transmission ofelectrical data signals to components such as processors 132, memories,sensors, input devices, output devices, and communication devices.Accordingly, the communication bus 120 may comprise a bus. Additionally,it is noted that the term “signal” means a waveform (e.g., electrical,optical, magnetic, mechanical or electromagnetic), such as DC, AC,sinusoidal-wave, triangular-wave, square-wave, vibration, and the like,capable of traveling through a medium. The communication bus 120communicatively couples the various components of the system 100. Asused herein, the term “communicatively coupled” means that coupledcomponents are capable of exchanging signals with one another such as,for example, electrical signals via conductive medium, electromagneticsignals via air, optical signals via optical waveguides, and the like.

The electronic control unit 130 may be any device or combination ofcomponents comprising a processor 132 and the memory component 134. Theprocessor 132 of the system 100 may be any device capable of executingthe machine-readable instruction set stored in the memory component 134.Accordingly, the processor 132 may be an electric controller, anintegrated circuit, a microchip, a field programmable gate array, acomputer, or any other computing device. The processor 132 iscommunicatively coupled to the other components of the system 100 by thecommunication bus 120. Accordingly, the communication bus 120 maycommunicatively couple any number of processors 132 with one another,and allow the components coupled to the communication bus 120 to operatein a distributed computing environment. Specifically, each of thecomponents may operate as a node that may send and/or receive data.While the embodiment depicted in FIG. 1 includes a single processor 132,other embodiments may include more than one processor 132.

The memory component 134 of the system 100 is coupled to thecommunication bus 120 and communicatively coupled to the processor 132.The memory component 134 may be a non-transitory computer readablememory and may comprise RAM, ROM, flash memories, hard drives, or anynon-transitory memory device capable of storing machine-readableinstructions such that the machine-readable instructions can be accessedand executed by the processor 132. The machine-readable instruction setmay comprise logic or algorithm(s) written in any programming languageof any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as machinelanguage that may be directly executed by the processor 132, or assemblylanguage, object-oriented programming (OOP), scripting languages,microcode, etc., that may be compiled or assembled into machine readableinstructions and stored in the memory component 134. Alternatively, themachine-readable instruction set may be written in a hardwaredescription language (HDL), such as logic implemented via either afield-programmable gate array (FPGA) configuration or anapplication-specific integrated circuit (ASIC), or their equivalents.Accordingly, the functionality described herein may be implemented inany conventional computer programming language, as pre-programmedhardware elements, or as a combination of hardware and softwarecomponents. While the system 100 depicted in FIG. 1 includes a singlememory component 134, other embodiments may include more than one memorycomponent 134.

Still referring to FIG. 1, in some embodiments, the system 100 mayinclude a LIDAR system 136. The LIDAR system 136 is communicativelycoupled to the communication bus 120 and the electronic control unit130. The LIDAR system 136 is used in a light detection and rangingsystem that uses pulsed laser light to measure distances from the LIDARsystem 136 to objects that reflect the pulsed laser light. The LIDARsystem 136 may be made of solid-state devices with few or no movingparts, including those configured as optical phased array devices whereits prism-like operation permits a wide field-of-view without the weightand size complexities associated with a traditional rotating LIDARsensor. The LIDAR system 136 is particularly suited to measuringtime-of-flight, which in turn can be correlated to distance measurementswith objects that are within a field-of-view of the LIDAR system 136. Bycalculating the difference in return time of the various wavelengths ofthe pulsed laser light emitted by the LIDAR system 136, a digital 3-Drepresentation of a target or environment may be generated. The pulsedlaser light emitted by the LIDAR system 136 may be operated in or nearthe infrared range of the electromagnetic spectrum, with one examplehaving emitted radiation of about 905 nanometers. Sensors such as LIDARsystem 136 can be used by vehicles such as vehicle 110 (FIG. 2) toprovide detailed 3-D spatial information for the identification ofobjects near the vehicle 110, as well as the use of such information inthe service of systems for vehicular mapping, navigation and autonomousoperations, especially when used in conjunction with geo-referencingdevices such as GPS or a gyroscope-based inertial navigation unit (INU,not shown) or related dead-reckoning system, as well as non-transitorycomputer readable memory 134 (either its own or memory of the electroniccontrol unit 130).

The system 100 may also include one or more cameras 138. The one or morecameras 138 may be communicatively coupled to the communication bus 120and to the processor 132. The one or more cameras 140 may be any devicehaving an array of sensing devices (e.g., pixels) capable of detectingradiation in an ultraviolet wavelength band, a visible light wavelengthband, or an infrared wavelength band. The one or more cameras 138 mayhave any resolution. The one or more cameras 138 may be anomni-directional camera, or a panoramic camera, for example. In someembodiments, one or more optical components, such as a mirror, fish-eyelens, or any other type of lens may be optically coupled to each of theone or more cameras 138. In embodiments described herein, the one ormore cameras 138 may capture image data or video data of an environmentof a vehicle. That is, with reference to FIG. 2, a vehicle 110 havingwheels 115 may include one or more cameras 138 mounted thereon. The oneor more cameras 138 may be mounted on a dashboard of a vehicle 110, on arearview mirror, or elsewhere on the vehicle 110 such that the one ormore cameras 138 may capture road-scene data of the environment around avehicle 110. In some embodiments, the vehicle 110 may further includeone or more components of the system 100 such as an electronic controlunit 130 configured with a trained neural network as described herein.That is, the vehicle 110 may receive road-scene data from the camera andpredict, with the neural network, an occurrence of a handback event. Inresponse to predicting an occurrence of a handback event, the electroniccontrol unit 130 may output an alert to the driver warning them of animpending handback event so that the driver may become aware of theenvironment of the vehicle and prepare to receive operational control ofthe vehicle 110.

The system 100 may include a gaze-tracking system 140 for tracking aneye or gaze direction of a subject to generate a gaze direction vectorfor determining where a driver is looking. The gaze-tracking system 140may include one or more cameras 138 and/or an array of infrared lightdetectors positioned to view one or more eyes of a subject. Thegaze-tracking system 140 may also include or be communicatively coupledto an illuminating device 141 which may be an infrared or near-infraredlight emitter. The illuminating device 141 may emit infrared ornear-infrared light, which may be reflected off a portion of the eyecreating a profile that is more readily detectable than visible lightreflections off an eye for eye-tracking purposes.

The gaze-tracking system 140 may be spatially oriented in an environmentand generate a gaze direction vector. One of a variety of coordinatesystems may be implemented such as user coordinate system (UCS). Forexample, the UCS has its origin at the center of the front surface ofthe gaze-tracker. With the origin defined at the center of the frontsurface (e.g., the eye-tracking camera lens) of the gaze-tracking system140, the gaze direction vector may be defined with respect to thelocation of the origin. Furthermore, when spatially orienting thegaze-tracking system 140 in the environment, all other objects includingthe one or more cameras 138 may be localized with respect to thelocation of the origin of the gaze-tracking system 140. In someembodiments, an origin of the coordinate system may be defined at alocation on the subject, for example, at a spot between the eyes of thesubject. Irrespective of the location of the origin for the coordinatesystem, a calibration process may be employed by the gaze-trackingsystem 140 to calibrate a coordinate system for collecting gaze-trackingdata for training the neural network.

Still referring to FIG. 1, the system 100 may further include one ormore physiological sensors 142. The one or more physiological sensors142 may be communicatively coupled to the communication bus 120 and tothe processor 132. The one or more physiological sensors 142 may be anydevice capable of monitoring and capturing physiological states of thehuman body, such as a driver's stress level through monitoringelectrical activity of the heart, skin conductance, respiration, or thelike. The one or more physiological sensors 142 include sensorsconfigured to measure bodily events such as heart rate change,electrodermal activity (EDA), muscle tension, and cardiac output. Theone or more physiological sensors 142 may monitor brain waves throughelectroencephalography, EEG, electrodermal activity through a skinconductance response, SCR, and galvanic skin response, GSR,cardiovascular measures such as heart rate, HR; beats per minute, BPM;heart rate variability, HRV; vasomotor activity, muscle activity throughelectromyography, EMG, changes in pupil diameter with thought andemotion through pupillometry (e.g., pupillometry data), eye movements,recorded via the electro-oculogram, EOG and direction-of-gaze methods,and cardiodynamics recorded via impedance cardiography, or otherphysiological measures.

The physiological sensors 142 may generate physiological response datathat may be utilized to train or evolve a neural network to determine astate of awareness of a driver. For example, a speed of change, thedegree of change, or the intensity of the resulting physiologicalcondition such as the speed or amount of pupil dilation or elevation inheart rate may be captured by the one or more physiological sensors 142.The observed changes may be translated into a state of awareness ofconditions within the environment.

The system 100 may also include a speaker 144. The speaker 144 (i.e., anaudio output device) is coupled to the communication bus 120 andcommunicatively coupled to the processor 132. The speaker 144 transformsaudio message data as signals from the processor 132 of the electroniccontrol unit 130 into mechanical vibrations producing sound. Forexample, the speaker 144 may provide to the driver a notification,alert, or warning of an impending handback event. The notification mayinclude prompts such as an estimate as to how much time until a handbackevent, information about the environment such as “entering aconstruction zone, prepare to assume control of the vehicle,” or otherinformation to alert the driver of a predicted handback event. However,it should be understood that, in other embodiments, the system 100 maynot include the speaker 144.

The steering wheel system 146 is coupled to the communication bus 120and communicatively coupled to the electronic control unit 130. Thesteering wheel system 146 may comprise a plurality of sensors located inthe steering wheel for determining a driver grip on the steering wheel,the degree of rotation applied to the steering wheel or the forcesapplied in turning or maintaining the steering wheel. The steering wheelsystem 146 may provide signals to the electronic control unit 130indicative of the location and number of hands on the steering wheel,the strength of the grip on the steering wheel, or changes in positionof one or more hands on the steering wheel. The steering wheel system146, for example, without limitation, may include pressure sensors,inductive sensors, optical sensors, or the like. In addition todetecting the location, number, grip and change in position of one ormore hands on the steering wheel, the steering wheel system 146 may alsoinclude one or more sensors indicating the rotational angle of thesteering wheel and corresponding signals to the electronic control unit130. As later described, the combination of steering wheel rotation andsteering wheel grip may be suggestive of a driver planning to ordesiring to take control of the vehicle. The steering wheel system 146may include motors or components to provide haptic feedback to thedriver. For example, the steering wheel system 146 may be configured toprovide vibrations of varying intensity through the steering wheel toindicate the varying likelihood that a predicted handback event willoccur.

The heads-up display system 148 may be included with the system 100 forpresenting visual indications to a driver of locations within theenvironment that are salient to the predicted handback event. Forexample, a heads-up display system 148 may highlight or annotatedportions of the environment having conditions that are related to aprediction of a handback event. Such indications may allow the driver torapidly ascertain conditions within the environment that are importantto become aware of in preparation of assuming operational control of thevehicle. A heads-up display system 148 may be a display deviceintegrated with the windshield or other display device within thevehicle. In some embodiments, the heads-up display system 148 mayinclude a projector that projects images onto the windshield through oneor more lens systems. However, this is only one example implementationof a heads-up display system 148.

The system 100, for example, as implemented in a vehicle 110 (FIG. 2),may include a vehicle display 149. The vehicle display 149 may be adisplay device. The display device may include any medium capable oftransmitting an optical output such as, for example, a cathode ray tube,light emitting diodes, a liquid crystal display, a plasma display, orthe like. The vehicle display 149 may be configured to display a visualalert or warning message, image data or a saliency map of the vehicleenvironment, or the like to the driver. The visualization on the vehicledisplay 149 may assist in bringing one or more portions of anenvironment to the driver's attention that may correspond to detectedconditions within the environment that correspond to a predictedhandback event. The vehicle display 149 may also include one or moreinput devices. The one or more input devices may be any device capableof transforming user contact into a data signal that can be transmittedover the communication bus 120 such as, for example, a button, a switch,a knob, a microphone or the like. In some embodiments, the one or moreinput devices include a power button, a volume button, an activationbutton, a scroll button, or the like. The one or more input devices maybe provided so that the user may interact with the vehicle display 149,such as to navigate menus, make selections, set preferences, and otherfunctionality described herein. In some embodiments, the input deviceincludes a pressure sensor, a touch-sensitive region, a pressure strip,or the like.

A data storage component 150 that is communicatively coupled to thesystem 100 may be a volatile and/or nonvolatile digital storagecomponent and, as such, may include random access memory (includingSRAM, DRAM, and/or other types of random access memory), flash memory,registers, compact discs (CD), digital versatile discs (DVD), and/orother types of storage components. The data storage component 150 mayreside local to and/or remote from the system 100 and may be configuredto store one or more pieces of data (e.g., driving data 152, environmentinformation 154, gaze patterns 156, known conditions 158, and/or salientportions 160 of the environment) for access by the system 100 and/orother components. As illustrated in FIG. 1, the data storage component150 stores, for example, driving data 152 that may include informationfrom one or more environment sensors recorded during past drivingevents. The driving data 152 may include image data, LIDAR data, speeddata, location information, navigation or route information,acceleration or deceleration activity, or the like. The driving data 152may be segmented into sets of data where a first set of driving data 152includes information leading up to an automatic transfer of operationalcontrol from an autonomous system to a human driver (i.e., a handbackevent). A second set of driving data 152 may include information leadingup to a manual transfer of operational control of a vehicle by thedriver. For example, the driver may manually deactivate an autonomoussystem and take over operation of the vehicle.

The data storage component 150 may also include environment information154. The environment information 154 includes information generated byone or more environment sensors. In some embodiments, the informationfrom the one or more environment sensors may be temporarily stored inthe data storage component 150 before processing by the electroniccontrol unit 130. While in some embodiments, the environment information154 is recorded for later analysis or use in training a neural networkto predict an occurrence of a handback event. The data storage component150 may also record and/or store gaze pattern data 156 generated by thegaze-tracking system 140.

In some embodiments, the system 100 may be preprogramed or include alearned set of known conditions 158 that occur or present themselves inan environment prior to a handback event. The one or more knownconditions 158 may be compared or analyzed in conjunction withenvironment information 154 to determine whether at least one conditionof possibly several conditions exist within the environment information154 that would lead to a handback event. For example, a first handbackevent may occur when known conditions: condition A, condition B, andcondition C are detected within the environment information 154.Therefore, a prediction that a handback event may occur may be made whencondition A is detected in the environment information 154. Thelikelihood that the predicted handback event occurs may include aconfidence value of 33% on a scale of 0% to 100% with 100% representingan imminent handback event. As further conditions are detected, forexample, once condition A and condition B are detected, then thelikelihood that the predicted handback event occurs may include anincrease confidence value (i.e., greater than 33% where only condition Awas detected). Moreover, once condition A, condition B, and condition Care detected, then the likelihood that the predicted handback eventoccurs may include an even further increase in the confidence value.

In some embodiments, image data captured by the one or more cameras 138may be analyzed to determine salient portions 160 within the image dataof the environment. The salient portions 160 correspond to the one ormore known conditions 158 that lead to a handback event. Salientportions 160 may be determined through implementation of one or moreknown or yet to be developed saliency based image segmentationalgorithms. As will be described in more detail herein, the salientportions 160 of the environment may be compared with a gaze pattern of adriver to determine that state of awareness of a driver.

Still referring to FIG. 1, the system 100 may also include networkinterface hardware 170 that is communicatively coupled to the electroniccontrol unit 130 via the communication bus 120. The network interfacehardware 170 may include any wired or wireless networking hardware, suchas a modem, LAN port, Wi-Fi card, WiMax card, mobile communicationshardware, and/or other hardware for communicating with a network 180and/or other devices and systems. For example, the system 100 may becommunicatively coupled to a network 180 by way of the network interfacehardware 170.

Turning now to FIG. 2, an illustrative vehicle 110 implemented with thesystem 100 is depicted. The system for predicting a handback event maybe implemented in a vehicle 110 having one or more wheels 115. Thevehicle 110 may include one or more environment sensors, for example,one or more cameras 138 mounted on the vehicle 110 and communicativelycoupled to an electronic control unit 130. The vehicle 110 may be alevel 1, level 2, level 3, level 4, or level 5 autonomous vehicle 110.

The vehicle 110 may also include a gaze-tracking system 140 formonitoring the state of awareness of a driver 205, as illustrativelydepicted in FIG. 3. The gaze-tracking system 140 may be positioned inthe vehicle 110 so that a camera or detection device (e.g., 138) isconfigured to capture eye, head, and/or body positions of the driver. Anilluminating device 141, for example, an infrared lamp, may directinfrared light toward the driver to enhance detection of eye, head,and/or body positions, which may translate into gaze direction vectors.

Referring now to FIG. 4, an illustrative diagram for training a neuralnetwork 400 for predicting the transfer of control of the vehicle to thedriver is depicted. In some embodiments, the neural network 400 mayinclude one or more layers 405, 410, 415, 420, having one or more nodes401, connected by node connections 402. The one or more layers 405, 410,415, 420 may include an input layer 405, one or more hidden layers 410,415, and an output layer 420. The input layer 405 represents the rawinformation that is fed into the neural network 400. For example,environment information 154 from one or more environment sensors (e.g.,the LIDAR system 136 and/or one or more cameras 138), known conditions158, gaze patterns 156 from the gaze-tracking system 140, physiologicalresponse data from one or more physiological sensors 142, and/or drivingdata 152 may be input into the neural network 400 at the input layer405. The neural network 400 processes the raw information received atthe input layer 405 through nodes 401 and node connections 402. The oneor more hidden layers 410, 415, depending on the inputs from the inputlayer 405 and the weights on the node connections 402, carry outcomputational activities. In other words, the hidden layers 410, 415perform computations and transfer information from the input layer 405to the output layer 420 through their associated nodes 401 and nodeconnections 402.

In general, when a neural network 400 is learning, the neural network400 is identifying and determining patterns within the raw informationreceived at the input layer 405. In response, one or more parameters,for example, weights associated to node connections 402 between nodes401, may be adjusted through a process known as back-propagation. Itshould be understood that there are various processes in which learningmay occur, however, two general learning processes include associativemapping and regularity detection. Associative mapping refers to alearning process where a neural network 400 learns to produce aparticular pattern on the set of inputs whenever another particularpattern is applied on the set of inputs. Regularity detection refers toa learning process where the neural network 400 learns to respond toparticular properties of the input patterns. Whereas in associativemapping the neural network 400 stores the relationships among patterns,in regularity detection the response of each unit has a particular‘meaning’. This type of learning mechanism may be used for featurediscovery and knowledge representation.

Neural networks possess knowledge that is contained in the values of thenode connection weights. Modifying the knowledge stored in the networkas a function of experience implies a learning rule for changing thevalues of the weights. Information is stored in a weight matrix W of aneural network. Learning is the determination of the weights. Followingthe way learning is performed, two major categories of neural networkscan be distinguished: 1) fixed networks in which the weights cannot bechanged (i.e., dW/dt=0) and 2) adaptive networks that are able to changetheir weights (i.e., dW/dt not=0). In fixed networks, the weights arefixed a priori according to the problem to solve.

In order to train a neural network to perform some task, adjustments tothe weights are made in such a way that the error between the desiredoutput and the actual output is reduced. This process may require thatthe neural network compute the error derivative of the weights (EW). Inother words, it must calculate how the error changes as each weight isincreased or decreased slightly. A back propagation algorithm is onemethod that is used for determining the EW.

The algorithm computes each EW by first computing the error derivative(EA), the rate at which the error changes as the activity level of aunit is changed. For output units, the EA is simply the differencebetween the actual and the desired output. To compute the EA for ahidden unit in the layer just before the output layer, first all theweights between that hidden unit and the output units to which it isconnected are identified. Then, those weights are multiplied by the EAsof those output units and the products are added. This sum equals the EAfor the chosen hidden unit. After calculating all the EAs in the hiddenlayer just before the output layer, in like fashion, the EAs for otherlayers may be computed, moving from layer to layer in a directionopposite to the way activities propagate through the neural network,hence “back propagation”. Once the EA has been computed for a unit, itis straightforward to compute the EW for each incoming connection of theunit. The EW is the product of the EA and the activity through theincoming connection. It should be understood that this is only onemethod in which a neural network is trained to perform a task.

Referring back to FIG. 4, the neural network 400 may include one or morehidden layers 410, 415 that feed into one or more nodes 401 of an outputlayer 420. There may be one or more output layers 420 depending on theparticular output the neural network 400 is configured to generate. Forexample, the neural network 400 may be trained to output a prediction ofan occurrence of a handback event 430, generate a confidence value 440associated with the predicted handback event 430, determine a degree andtype of an alert 450 for providing to the driver and/or determiningknown conditions 460 that leads to a handback event 430. The knownconditions 460 that are determined by the neural network 400 in trainingmay be used as feedback for further training the neural network 400.

Turning to FIGS. 5A-5C, an illustrative flow diagram 500 for predictinga transfer of control of a vehicle to a driver is depicted. Flow diagram500 is only one method, implemented by a system that predicts a transferof control from an autonomous mode to a driver. At block 502, the systemreceives information about an environment of the vehicle. Theinformation includes road scene data in the form of image data, LIDARdata, and/or another form of data that can be analyzed to determineconditions within the environment from the information collected by oneor more of the environment sensors. At block 504, the system identifiesat least one condition represented in the information that correspondsto at least one of one or more known conditions that leads to a handbackof operational control of the vehicle to the driver.

In some embodiments, the system may receive driving data, which includesdata from one or more driving events, at block 505. The system analyzesthe driving data. The driving data includes data from one or moredriving events where a handback event occurred or the driver manuallyassumed control of a vehicle. The driving events include informationabout the environment around the vehicle during the time leading up tothe handback event and/or during the handback event. At block 506, thedriving data may be analyzed, compared with other driving data, and/orprocessed by other means (e.g., by a neural network) to define one ormore known conditions that are present in an environment during apredefined period of time leading up to a handback event and/or duringthe handback event. For example, the driving data may be analyzed toidentify common conditions in an environment that occur or presentthemselves prior to a handback event. The one or more known conditionsmay form a set of conditions and/or a sequence of events that presentthemselves prior to a handback event occurring.

For example, a sequence of events including a plurality of knownconditions may include: determining that the vehicle is in aconstruction zone, determining that there will be a change in trafficpattern due to the construction zone, and determining the presence ofcones, a flagman, or other indicator that the traffic pattern in beingaltered to traverse the constructions zone. In some instances, when avehicle is operating in an autonomous mode or is equipped withautonomous driving mechanisms, the autonomous system may be configuredto handback control of a vehicle within a construction zone becausetraversing a construction zone may include various obstacles oractivities that would be better suited for traversal by a human driveror at least overseen by a human driver should intervention be needed.Traversing a construction zone is only one example of an environmentwhere a handback event may occur. Other non-limiting examples ofenvironments where a handback event may occur include traversing schoolzones, shopping mall streets, parking lots, unmarked roads, accidentscenes, traffic intense roads, or roads during certain weather events(e.g., snow storms, rain, fog, and/or high winds). Other drivingenvironments may require a human driver to assist or take full controlof maneuvering the vehicle through the environment. Such environmentsmay depend on the sophistication of the autonomous driving system and/orthe types and operational status of the various sensors equipped on thevehicle.

Referring back to block 504, the one or more known conditions determinedfrom driving data may be utilized to compare, analyze, or otherwiseprocess real-time, near real-time, or stored information obtained by theone or more environment sensors during driving to identify at least onecondition that corresponds to a known condition present in theenvironment. At block 508, the system predicts an occurrence of ahandback event based on the at least one condition identified from theinformation about the environment of the vehicle. As discussed in moredetail below, the prediction may include a confidence valuecorresponding to a likelihood that the predicted occurrence of thehandback event will occur and/or determining and highlighting salientportions of an environment to a driver.

In some embodiments, at block 510 (See FIG. 5B, which illustrates thecontinuation of the flow diagram 500 of FIG. 5A), the system determineswhether the information includes image data. In the event image data isincluded in the information about the environment of the vehicle, thesystem, at block 512, determines one or more salient portions of theenvironment that corresponds to the identified conditions. FIGS. 6A-6C,described in further detail below provide, an illustrative example ofdetermining the one or more salient portions of the environment. Thesystem may utilize one or more known yet to be developed saliency basedimage segmentation algorithms, object detection algorithms, and/or otherimage analysis approaches to determine the one or more salient portionsof the environment.

The system may further receive gaze direction vectors from a gazetracking system configured to monitor the gaze of a driver at block 514.At block 516, the gaze direction vectors may be processed to determine agaze pattern of the driver. The gaze pattern may be visually representedas a heat map or gaze plot overlaid with the image data of theenvironment of the vehicle thereby identifying where the driver's gazeis concentrated. At block 518, the system compares the gaze pattern datato the one or more salient portions of the environment previouslydetermined. The comparison carried out at block 518 results in adetermination of whether the driver is or has gazed upon salientportions of the environment. In other words, the system may determinewhether the driver is aware of the conditions in the environment, whichhave been identified as leading to a handback event and likewise apredicted occurrence of the handback event. Moreover, the system, atblock 520, can determine a state of awareness of the driver from thecomparison carried out in block 518. The state of awareness indicateswhether the driver is paying attention to the environment of the vehicleand whether the driver is aware or at least gazed upon a feature in theenvironment corresponding to an identified and known condition that canlead to a handback event.

In response to determining the salient portions in the environment, thegaze pattern of a driver, and the state of awareness of the driver, thesystem determines an appropriate type and degree of an alert may bedetermined at block 522. For example, an alert may be a visual,auditory, haptic, combination thereof or another type. The degree mayrange from subtle to intense. A subtle alert may include presentinginformation on a display or verbally communicating with the driver thata condition leading to a handback event is detected. The degree of thealert may increase as the prediction that a handback event will occurbecomes more likely. The degree of the alert may define the intensity,the amount of detail provided, or the interval of an auditory alert.Similarly, the degree of a visual type of alert may define the intensityand/or interval of visual indictors or displayed information configuredto attain the attention of the driver and/or the amount of detailprovided via the visual alert. A haptic type of alert may include avarying degree of force feedback and/or vibrations.

At block 524, once an alert is determined, the system provides the alertto the driver. In some embodiments, the alert may be a visual presentedon a heads-up display highlighting salient portions of the environmentto the driver in order to attain the driver's attention and make heraware of the one or more conditions identified and related to apredicted handback event. In some embodiments, a combination of alertsmay be determined and generated. The alerts are configured to notify thedriver with a type and degree of alert that corresponds to the imminenceand/or likelihood that the predicted handback event occurs. That is, bygrading the alert in such a way that corresponds to the driver'sawareness, the driver may not become desensitized or ignore overlyintense alerts when the predicted handback event is perceived as notlikely by the driver or the driver is already prepared to receiveoperational control of the vehicle. Moreover, by combining the state ofawareness of the driver with the determination of the degree and/or typeof alert, a lesser intense alert may be needed if the driver is alreadyaware of a condition in an environment.

Referring back to block 510, if no image data is included in theinformation, then the process continues to block 530, as shown in FIG.5C. At block 530, the system determines a confidence value correspondingto the likelihood that the predicted occurrence of the handback eventwill occur. The confidence value may be determined based on the numberof conditions identified in the information about the environment aroundthe vehicle, a weighting associated with the conditions, the occurrenceof a sequence of conditions identified in the information and/or basedon other methods of assigning a confidence value to a prediction. Insome embodiments, as more conditions that correspond to known conditionsthat lead to a handback event are identified in environments around thevehicle based on the received information, a confidence interval may beincreased. For example, assume a handback event is known to occur whenknown conditions: condition A, condition B, and condition C are presenteither in combination or in a predefined sequence within an environment.When condition A is identified from the information about theenvironment, then the confidence value may be determined to be at afirst value (e.g., indicating that the likelihood the handback eventwill occur is 33%). When condition B is also identified then theconfidence value may increase to a second value (e.g., 66%). Whencondition C is further identified then the confidence value may increaseto a third value (e.g., 90%).

In some embodiments, the order in which the known conditions occurand/or are identified in the information about the environment aroundthe vehicle may affect the determined confidence value. For example, ifa handback event is known to occur more often when known conditions A, Band C occur in order then when, for example, condition A is identifiedand then condition C is identified in the information, the confidencevalue may not be as high as a confidence value where conditions A, B andthen C were identified. In such a case, the handback event may stilloccur when only conditions A and then C are identified because conditionB may not have been detected, or may not be necessary for the specifichandback event to occur. In some embodiments, each of the conditionsleading up to a handback event may culminate in a single trigger eventthat causes the system to handback control of the vehicle or takealternative action if the driver is not ready to assume control. Thetrigger event may also be considered a condition and may result in aprediction having a confidence interval of, for example 100%.

The previous examples described herein refer generally to conditionswithout specific reference to what may constitute conditions leading toa handback event. In some embodiments, the conditions may includeidentifying a road sign, detecting an object, detecting a pattern ofmotion of traffic that may be abnormal to the expected flow, detectingthe presence of flashing lights (e.g., construction yellow lights, redor blue emergency lights, or the like), detecting from the environmentsensors a weather event such as rain or snow, or any other set offeatures or events that an autonomous system may be preprogrammed totransfer control back to a driver should they be present or occur. Theexamples herein are non-limiting and that the systems and methodsdescribed herein may be configured to identify any number and variety ofconditions leading to a handback event and subsequently predict theoccurrence of a handback event.

Still referring to FIGS. 5A-5C, once the confidence value is determinedat block 530, the system determines a type and/or degree of alert toprovide to a driver based on the confidence value at block 532. Asdescribe above with reference to block 522, the alert may be a visual,auditory, haptic, combination thereof or another type. The degree mayrange from subtle to intense. A subtle alert may include presentinginformation on a display or verbally communicating with the driver thata condition leading to a handback event is detected. The degree of thealert may increase as the confidence value associated with theprediction that a handback event will occur increases. The degree of thealert may define the intensity, the amount of detail provided, or theinterval of an auditory alert. Similarly, the degree of a visual type ofalert may define the intensity and/or interval of visual indictors ordisplayed information configured to attain the attention of the driverand/or the amount of detail provided via the visual alert. A haptic typeof alert may include a varying degree of force feedback and/orvibrations.

At block 534, once an alert is determined, the system provides the alertto the driver. In some embodiments, the alert may be an auditory orvisual alert that increases in intensity as the confidence valueincreases. The alert may be provided through a speaker, a visualdisplay, an illumination device, or the like. The alert may also beprovided as a haptic alert. For example, the driver seat or steeringwheel may be configured to vibrate to alert the driver of a predictedhandback event. In some embodiments, a combination of alerts may bedetermined and generated. The alerts are configured to notify the driverwith a type and degree of alert that corresponds to the imminence and/orlikelihood that the predicted handback event occurs. That is, by gradingthe alert in such a way that it corresponds to the confidence value ofthe prediction, the driver may not become desensitized or ignore overlyintense alerts when the predicted handback event is perceived as notlikely by the driver.

Turning now to FIGS. 6A-6C, an illustrative environment 600 around avehicle 110 is depicted where several conditions that lead to a handbackevent are present. In particular, FIG. 6A depicts an illustrativeenvironment 600 around a vehicle 110 including one or more conditionsindicative of a transfer of control of the vehicle to the driver,according to one or more embodiments shown and described herein. Forexample, and without limitation, an environment 600 may include avehicle 110 traversing a construction zone along a roadway 601. Theconstruction zone may include a construction sign 602 that alerts adriver to the presence of a beginning of a construction zone. Forexample, the construction sign 602 may include a message such as “ROADWORK AHEAD.” The construction environment 600 may also include a trafficsign 603, which alerts a driver to a change in traffic patterns as aresult of the construction. For example, a traffic sign 603 may includea message such as “LANE ENDS” that alerts the driver to the closureand/or merger of one or more lanes. The construction environment 600 mayalso include a plurality of cones or construction barrels 604 that actas physical barriers to a vehicle forcing a vehicle to change their pathor otherwise colliding with the construction barrels 604. Each of thesesigns and objects may serve as a known condition that indicates and/orleads to a handback event.

FIG. 6B depicts the same illustrative environment 600 from FIG. 6A butnow includes salient portions identified by the systems and methodsdescribed herein that correspond to the one or more known conditionsindicative of a transfer of control of the vehicle to the driver. Forexample, the system and method may employ one or more saliencytechniques and/or algorithms that identify the salient portions in theimage data that correspond to the conditions that may lead to a handbackevent. For example, a first salient portion 622 may be determined by thesystem, as depicted in FIG. 6B, encompassing the construction sign 602.Additionally, a second salient portion 623 may be determined by thesystem, as depicted in FIG. 6B, that encompasses the traffic sign 603.Furthermore, a third salient portion 624 may be determined by thesystem, as depicted in FIG. 6B, that encompasses one or more of thecones or construction barrels 604. Referring back to the methoddescribed in FIGS. 5A-5C, the salient portions of the image data of theenvironment 600 may be compared with gaze pattern data which may bepresentable as a heat map or gaze plot, as illustratively depicted inFIG. 6C, to determine whether the driver is aware of a condition in theenvironment 600 that leads to a handback event.

Turning to FIG. 6C, an illustrative environment 600 around a vehicleincluding gaze pattern of a driver is depicted. Based on a gaze trackingsystem, a gaze pattern and subsequent heat map and/or gaze plot may berendered which corresponds to the gazing activity of a driver. Forexample, as depicted in FIG. 6C, a driver appears to gaze in generallythree areas of the vehicle environment 600 over a predefined period oftime (e.g., over a few second, 10 seconds, 20 seconds, 30 seconds,etc.). The driver's gaze, for example, includes viewing the constructionsign 602 as indicated by the first illustrated gaze pattern 632, theroadway 601 generally in front of the vehicle as indicated by the secondillustrated gaze pattern 636, and the dashboard as indicated by thethird illustrated gaze pattern 638.

By way of a non-limiting example, when the gaze pattern illustration ofFIG. 6C is compared with the salient portions of FIG. 6B it may bedetermined that the driver is aware of the construction sign 602, butnot the traffic sign 603 or the construction barrels 604. As a result,when a handback event is predicted based on at least the identificationof the construction sign 602, the alert may subtle or optionally notprovided since the driver is aware of the first condition. However, asthe vehicle 110 continues to traverse the construction environment 600and the traffic sign 603 and/or the construction barrels 604 areidentified the provided alert may have an increased intensity or a moreacute type of alert may be employed to bring the conditions to thedriver's attention. That is, the driver may be made aware of theenvironment 600 so they may receive operational control of the vehiclewhen the handback is necessary.

It should be understood that the embodiments described herein aredirected to systems and methods for predicting occurrences of a handbackevent. In embodiments, the systems and methods may utilize an electroniccontrol unit and/or neural network to receive information about anenvironment of the vehicle, identify at least one condition representedin the information about the environment of the vehicle that correspondsto at least one of one or more known conditions that lead to a handbackof operational control of the vehicle to the driver, and predict anoccurrence of a handback event based on the at least one conditionidentified from the information about the environment of the vehicle. Inembodiments that include a neural network, the neural networks may betrained in a variety of ways with a variety of data inputs. Theselection of such data inputs for training may correspond to theultimate or desired implementation of the neural network. That is, theneural network may be trained using data that will be available to thenetwork when the trained neural network is implemented within a systemor vehicle.

It is noted that the terms “substantially” and “about” may be utilizedherein to represent the inherent degree of uncertainty that may beattributed to any quantitative comparison, value, measurement, or otherrepresentation. These terms are also utilized herein to represent thedegree by which a quantitative representation may vary from a statedreference without resulting in a change in the basic function of thesubject matter at issue.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

1. A method for predicting a transfer of control of a vehicle to adriver, the method comprising: receiving information about anenvironment of the vehicle; identifying at least one conditionrepresented in the information about the environment of the vehicle thatcorresponds to at least one of one or more known conditions that lead toa handback of operational control of the vehicle to the driver; andpredicting the transfer of control of the vehicle to the driver based onthe at least one condition identified from the information about theenvironment of the vehicle.
 2. The method of claim 1, wherein theinformation about the environment of the vehicle includes image data ofthe environment around the vehicle captured by a camera.
 3. The methodof claim 2, further comprising determining, from the image data, one ormore salient portions of the environment, wherein at least one of theone or more salient portions of the environment corresponds to the atleast one condition identified from the information about theenvironment of the vehicle.
 4. The method of claim 3, furthercomprising: determining a gaze pattern of the driver, wherein the gazepattern defines a heat map or a gaze plot identifying one or morelocations in the environment where the driver has gazed upon; comparingthe gaze pattern of the driver to the one or more salient portions ofthe environment determined from the image data; and determining a stateof awareness of the driver based on the comparison of the gaze patternand the one or more salient portions of the environment, wherein thestate of awareness defines a degree of awareness of the driver to the atleast one condition identified from the information about theenvironment of the vehicle.
 5. The method of claim 4, further comprisingproviding an alert to the driver, wherein a degree or type of the alertprovided to the driver corresponds to the state of awareness of thedriver.
 6. The method of claim 5, wherein the alert includes at leastone of the following: a projection on a heads-up display identifying atleast one of the one or more salient portions of the environment; anaudio notification; a visual indicator; or a haptic feedback.
 7. Themethod of claim 1, further comprising determining a confidence valuecorresponding to a likelihood that the predicted transfer of control ofthe vehicle to the driver occurs.
 8. The method of claim 7, furthercomprising providing an alert to the driver, wherein a degree or a typeof the alert provided corresponds to the confidence value.
 9. The methodof claim 1, further comprising: receiving driving data from one or moredriving events, wherein the driving data includes the information aboutthe environment of the vehicle leading up to and during past handbackevents; and determining the one or more known conditions that lead tothe handback of operational control of the vehicle to the driver. 10.The method of claim 9, wherein the driving data further includes theinformation about the environment of the vehicle when the drivermanually assumes control of the vehicle.
 11. A system for predicting atransfer of control of a vehicle to a driver comprising: an electroniccontrol unit; and one or more environment sensors communicativelycoupled to the electronic control unit, wherein the one or moreenvironment sensors capture information about an environment of thevehicle, wherein the electronic control unit is configured to: receivethe information about the environment of the vehicle from the one ormore environment sensors; identify at least one condition represented inthe information about the environment of the vehicle that corresponds toat least one of one or more known conditions that lead to a handback ofoperational control of the vehicle to the driver; and predict thetransfer of control of the vehicle to the driver based on the at leastone condition identified from the information about the environment ofthe vehicle.
 12. The system of claim 11, wherein the one or moreenvironment sensors includes a camera, wherein the information about theenvironment includes image data of the environment captured by thecamera, and wherein the electronic control unit is further configuredto: determine, from the image data, one or more salient portions of theenvironment, wherein at least one of the one or more salient portions ofthe environment corresponds to the at least one condition identifiedfrom the information about the environment of the vehicle.
 13. Thesystem of claim 12, further comprising a gaze tracking systemcommunicatively coupled to the electronic control unit, wherein theelectronic control unit is further configured to: receive gaze directionvectors of the driver from the gaze tracking system; determine a gazepattern of the driver, wherein the gaze pattern defines a heat map or agaze plot identifying one or more locations in the environment where thedriver has gazed upon; compare the gaze pattern of the driver to the oneor more salient portions of the environment determined from the imagedata; and determine a state of awareness of the driver based on thecomparison of the gaze pattern and the one or more salient portions ofthe environment, wherein the state of awareness defines a degree ofawareness of the driver to the at least one condition identified fromthe information about the environment of the vehicle.
 14. The system ofclaim 13, wherein the electronic control unit is further configured to:provide an alert to the driver, wherein a degree or type of the alertprovided to the driver corresponds to the state of awareness of thedriver.
 15. The system of claim 11, wherein the electronic control unitis further configured to: determine a confidence value corresponding toa likelihood that the predicted transfer of control of the vehicle tothe driver occurs.
 16. The system of claim 15, wherein the electroniccontrol unit is further configured to: provide an alert to the driver,wherein a degree or a type of the alert provided corresponds to theconfidence value.
 17. The system of claim 11, wherein the electroniccontrol unit is further configured to: receive driving data from one ormore driving events, wherein the driving data includes the informationabout the environment of the vehicle leading up to and during pasthandback events; and determine the one or more known conditions thatlead to the handback of operational control of the vehicle to thedriver.
 18. The system of claim 17, wherein the driving data furtherincludes the information about the environment of the vehicle when thedriver manually assumes control of the vehicle.
 19. A system forpredicting a transfer of control of a vehicle to a driver comprising: anelectronic control unit configured to implement a neural network; andone or more environment sensors communicatively coupled to theelectronic control unit, wherein the one or more environment sensorscapture information about an environment of the vehicle, and wherein theelectronic control unit is configured to: receive, as an input to theneural network, the information about the environment of the vehiclefrom the one or more environment sensors; identify, with the neuralnetwork, at least one condition represented in the information about theenvironment of the vehicle that corresponds to at least one of one ormore known conditions that lead to a handback of operational control ofthe vehicle to the driver; and predict, with the neural network, thetransfer of control of the vehicle to the driver based on the at leastone condition identified from the information about the environment ofthe vehicle.
 20. The system of claim 19, wherein the neural network istrained by: receiving driving data from one or more driving events,wherein the driving data includes the information about the environmentof the vehicle leading up to and during past handback events; analyzingthe driving data for common conditions leading up to past handbackevents; and determining the one or more known conditions that lead tothe handback of operational control of the vehicle to the driver.